EP 1 720 131 B1 shows an augmented reality system with real marker object identification. The system comprises a video camera for gathering image data from a real environment. The real environment represents any appropriate area, such as a room of a house, a portion of a specific landscape, or any other scene of interest. The real environment represents a living room comprising a plurality of real objects for instance in the form of walls and furniture. Moreover, the real environment comprise further real objects that are considered as marker objects which have any appropriate configuration so as to be readily identified by automated image processing algorithms. The marker objects have formed thereon significant patterns that may easily be identified, wherein the shape of the marker objects may be designed so as to allow identification thereof from a plurality of different viewing angles. The marker objects also represent substantially two-dimensional configurations having formed thereon respective identification patterns.
The system further comprises a means for identifying the marker objects on the basis of image data provided by the camera. The identifying means may comprise well-known pattern recognition algorithms for comparing image data with predefined templates representing the marker objects. The identifying means may have implemented therein an algorithm for converting an image obtained by the camera into a black and white image on the basis of predefined illumination threshold values. The algorithm are further configured to divide the image into predefined segments, such as squares, and to search for pre-trained pattern templates in each of the segments, wherein the templates represent significant portions of the marker objects.
First the live video image is turned into a black and white image based on a lighting threshold value. This image is then searched for square regions. The software finds all the squares in the binary image, many of which are not the tracking markers, such as the objects. For each square, the pattern inside the square is matched against some pre-trained pattern templates. If there is a match, then the software has found one of the tracking markers, such as the objects. The software then use the known square size and pattern orientation to calculate the position of the real video camera relative to the physical marker such as the objects. Then, a 3×4 matrix is filled with the video camera's real world coordinates relative to the identified marker. This matrix is then used to set the position of the virtual camera coordinates. Since the virtual and real camera coordinates are the same, the computer graphics that are drawn precisely superimpose the real marker object at the specified position. Thereafter, a rendering engine is used for setting the virtual camera coordinates and drawing the virtual images.
The system further comprises means for combining the image data received from the camera with object data obtained from an object data generator. The combining means comprise a tracking system, a distance measurement system and a rendering system. Generally, the combining means is configured to incorporate image data obtained from the generator for a correspondingly identified marker object so as to create virtual image data representing a three-dimensional image of the environment with additional virtual objects corresponding to the marker objects. Hereby, the combining means is configured to determine the respective positions of the marker objects within the real environment and also to track a relative motion between the marker objects with respect to any static objects in the environment and with respect to a point of view defined by the camera.
The system further comprises output means configured to provide the virtual image data, including the virtual objects generated by the generator wherein, in preferred embodiments, the output means is also configured to provide, in addition to image data, other types of data, such as audio data, olfactory data, tactile data, and the like. In operation, the camera creates image data of the environment, wherein the image data corresponds to a dynamic state of the environment which is represented by merely moving the camera with respect to the environment, or by providing moveable objects within the environment, for instance the marker objects or one or more of the objects are moveable. The point of view of the environment is changed by moving around the camera within the environment, thereby allowing to observe especially the marker objects from different perspectives so as to enable the assessment of virtual objects created by the generator from different points of view.
The image data provided by the camera which are continuously updated, are received by the identifying means, which recognizes the marker objects and enables the tracking of the marker objects once they are identified, even if pattern recognition is hampered by continuously changing the point of view by, for instance, moving the camera or the marker objects. After identifying a predefined pattern associated with the marker objects within the image data, the identifying means inform the combining means about the presence of a marker object within a specified image data area and based on this information, the means then continuously track the corresponding object represented by the image data used for identifying the marker objects assuming that the marker objects will not vanish over time. The process of identifying the marker objects is performed substantially continuously or is repeated on a regular basis so as to confirm the presence of the marker objects and also to verify or enhance the tracking accuracy of the combining means. Based on the image data of the environment and the information provided by the identifying means, the combining means creates three-dimensional image data and superimposes corresponding three-dimensional image data received from the object generator, wherein the three-dimensional object data are permanently updated on the basis of the tracking operation of the means.
The means may, based on the information of the identifying means, calculate the position of the camera with respect to the marker objects and use this coordinate information for determining the coordinates of a virtual camera, thereby allowing a precise “overlay” of the object data delivered by the generator with the image data of the marker objects. The coordinate information also includes data on the relative orientation of the marker objects with respect to the camera, thereby enabling the combining means to correctly adapt the orientation of the virtual object. Finally, the combined three-dimensional virtual image data is presented by the output means in any appropriate form. The output means may comprise appropriate display means so as to visualize the environment including virtual objects associated with the marker objects. When operating the system it is advantageous to pre-install recognition criteria for at least one marker object so as to allow a substantially reliable real-time image processing. Moreover, the correlation between a respective marker object and one or more virtual objects may be established prior to the operation of the system or is designed so as to allow an interactive definition of an assignment of virtual objects to marker objects. For example, upon user request, virtual objects initially assigned to the marker object are assigned to the marker object and vice versa. Moreover, a plurality of virtual objects is assigned to a single marker object and a respective one of the plurality of virtual objects is selected by the user, by a software application.
“Design and testing of a augmented reality head-up display in a vehicle” (German title: Entwicklung and Erprobung eines kontaktanalogen Head-up-Displays im Fahrzeug), M. Schneid, Dissertation, 2009, 2.1.2 Optical System, page 6 and 7 shows that the light beam of a head-up display are reflected by the windshield or a separate combiner into the driver's eyes. The size of the optical elements define the size of the light beam and the size fo the eyebox, that means the area in the y-z-plane, the eyes of the driver have to be positioned in to recognize the projected image. Having a bearing of the last mirror the reflection area on the windshield and the position of the eyebox can be adjusted.
The U.S. Pat. No. 5,214,413 shows a head-up display apparatus used as an instrument display of an automobile that comprises a hologram combiner provided on the front windshield on which an image is displayed overlapping the field of view. The apparatus includes a display luminosity control system which varies the brightness of the display image gradually or delays an increase and decrease in the brightness in accordance with variations in the ambient light level so as to allow a driver's eyes to adapt to the variations.
The object of the invention is to improve a system for a motor vehicle.
This object is attained by a system with the features of independent claim 1. Advantageous refinements are the subject of dependent claims and included in the description.
Therefore a system for a vehicle is provided. The system may be part of an infotainment system of the vehicle.
The system has a head-up display and a central unit connected to the head-up display.
The head-up display is configured to project an image onto the front windshield of the vehicle or onto a separate combiner.
The central unit is configured to send image data to the connected head-up display to be displayed.
The central unit is configured to ascertain a user's point of view. The user's point of view is the position of the user's eyes.
The central unit is configured to output a symbol within the image data.
The central unit is configured to ascertain a virtual point in the surrounding of the vehicle based on a recorded image of the surrounding and/or a current position of the vehicle. The virtual point in the surrounding may concurrently calculated. The virtual point in the surrounding may be calculated based on image data and/or map data.
The central unit is configured to align in the view of the user at least one point of the symbol to the virtual point. The virtual point in the surrounding may be overlaid in the view of the user e.g. by a dot of the symbol in the image displayed. The alignment is based on the user's point of view.
Tests by the applicant have shown that the route guidance using simple symbols are often misleading the driver, if the density of intersections is locally high. Using the augmented reality the symbols may point directly into the corresponding road. A possible movement of the head of the user is taken into account, resulting in a very precise guidance.
Another object of the invention is to improve a method for controlling a displayed image on a front windshield or on a separate combiner.
Therefore a method to control a displayed image on a front windshield of the vehicle or on a separate combiner is provided. The method comprises the steps:                projecting an image onto the front windshield of the vehicle or onto the combiner by means of a head-up display,        sending image data by means of a central unit to the connected head-up display to be displayed,        ascertaining a user's point of view by means of the central unit, the user's point of view being the position of the user's eyes,        outputting a symbol within the image data by means of the central unit,        ascertaining a virtual point in the surrounding of the vehicle based on a recorded image of the surrounding and/or a current position of the vehicle by means of the central unit, and        aligning in the view of the user at least one point of the symbol to the virtual point by means of the central unit, wherein the alignment is based on the user's point of view.        
The embodiments described hereinafter refer to both the system and the method.
According to one embodiment, the central unit may be configured to calculate the alignment of the at least one point of the symbol based on geometrical optics and/or trigonometric functions.
According to one embodiment, the infotainment system may have adjustment means for adjusting the position of the projected image within the plane of the front windshield or the combiner. The central unit may be configured to ascertain the user's point of view based on parameters of the adjustment of the projected image.
According to one embodiment, the infotainment system may have adjustment means for adjusting the position of a user's seat. The central unit may be configured to ascertain the user's point of view based on parameters of the adjustment of the user's seat.
According to one embodiment, the infotainment system may have an internal camera recording an image of the user. The central unit may be configured to recognize the user's eyes or the user's head. The central unit may be configured to ascertain the position of the user's eyes or the user's head within the recorded image of the user.
According to one embodiment, the central unit may be configured to track the position of the user's eyes or the user's head concurrently.
According to one embodiment, the central unit may be configured to align the position of the at least one point of the symbol to the virtual point concurrently based on at least one of a shift of the virtual point and a shift of the position of the user's eyes or the user's head.
According to one embodiment, the infotainment system may have a capture device recording an image of the surrounding. The central unit may be configured to ascertain a three dimensional space of the surrounding based on image data of the surrounding. The central unit may be configured to recognize an object within the image of the surrounding. The recognized object may have the virtual point in the three dimensional space. The central unit may be configured to align the at least one point of the symbol to the virtual point of the recognized object based on the position of the virtual point of the recognized object and the user's point of view.
According to one embodiment, the central unit may be configured to ascertain a three dimensional space of the surrounding based on map data of the surrounding. The central unit may be configured to recognize an object within the map data of the surrounding. The recognized object may have the virtual point in the three dimensional space. The central unit may be configured to align the at least one point of the symbol to the virtual point of the recognized object based on the position of the virtual point of the recognized object and the user's point of view.
According to one embodiment, the central unit may be configured to change the shape and/or the transparency and/or the colour of the symbol based on the user's point of view.
The previously described embodiments are especially advantageous both individually and in combination. In this regard, all embodiments may be combined with one another. Some possible combinations are explained in the description of the exemplary embodiments shown in the figures. These possible combinations of the refinement variants, depicted therein, are not definitive, however.